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The Effects of Fonmal Reasoning Task*s Characteristics 
on the Responses of Different Age Group Students 

Abstract 

The study examines the effects of Piagetian-like tasks» characteristics on 
the performance of these tasks by different age group students. The tasks 
were taken fVom a formerly developed and validated test, which measures 
students' reasoning skills in six cognitive operations: conservation, 
proportions, control of variables, probability, combinations and 
correlations. Subjects were 7th, 9th and 12th grade students, enrolled in 
two urban schools. 

Three different 3x2 factorial research design experiments, with three 
levels of students age and two versions of the test in each experiment, 
were set up for this study. Experiment 1 tested the effect of the method 
of task presentation (video-taped demonstrations versus pa per-and- pencil 
tasks with illustrations). Experiment 2 tested the effect of questionnaire 
format (multiple-choice versus short essay answers) and experiment 3 tested 
the effect of the numerical content (integer ratio like 1:2, 1:3 versus 
noninteger ratio like 2:3, 3:5) on different age group students* responses. 
Analysis of the data indicated that method of task presentation had effect 
only on young students performance, while numerical content had effect on 
the majority of the students, in all grades. The only students which were 
indifferent to numerical content changes were the formal reasoner ones. 
The implication of these findings to science c ucation in general, and to 
cognitive leval assessment in particular, ore discussed. 
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The Effects of Formal Reasoning Task's Characteristics 
on the Responses of Different Age Group Students 

I n t r 0 d u c t i o n 

In the last decade, group tests for assessing students cognitive levels 
took the place of the Piagetian clinical interviews. Some of these tests 
are Just paper-and-pencil tests (Longeot, 1965; Raven, 1973; Renner, 1979) > 
some are based on real demonstrations (Lawson, 1978; Shayer & Wharry, 197^0 
and others are based on video-taped demonstrations (Staver & Gabel, 1976; 
Tubin & Capie, 1981; Shemesh & Lazarowitz, 1984). The use of different 
methods to identify students* reasoning levels may cause the situation of 
contradictory findings in different studies, and by that reduce the ability 
of drawing meaningful conclusion from students performance on these tests. 
Staver (1981) for example, found that format of assessment in 
Piagetian-like reasoning tasks, but not method of presentation, had effect 
on college students' level of performance. In another study with 8th grade 
stuents, Staver (1985) found that the format of the questionnaire had no 
effect on students' level of performance. 

Wollman (1982) while testing the influence of tasks' content on student 
performance, found out that content can inhibit reasoning if it is too 
familiar. Studying the same issue. Linn et al (1982) found that familiar 
content cause students to adopt formal reasoning strategies more easily. 
The use of different tests is not the only source for the unclarity of 
these issues. Another possible source is that in each study different age 
group students were tested. Working memory and information processing 
capacity are still developing within the years of childhood to adolescence 
(Pascual-Leone and Goodman, 1979)* Since information processing is a 
mediating variable between task's characteristics and students' 
performance, students' age should be considered, while studying this iasue. 



Thus, the goal of this study was to investigate the effects which method of 
task presentation, format of test questionnaire and the nimierical content 
of the tasks, may have on the performance of different age group students. 
Vfhile in other studies, students* age was kept constant, in this study the 
age served as independent variable in order to find out the effect of each 
of the above mentioned factors separately. No attempt was made, in this 
study, to find the possible interactional effects of tasks* variables. 

Research Desig n 

Samplet The Sample consisted of 556 students from 7th, 9th and 12th grade, 
which were enrolled in urban schools, located in middle class environment. 
All students in the same grade level cover a similar curricula which 
includes one course in mathematics and at least one course of science per 
year, for every year of the secondary school. 

Instrumentst The tasks, used in this study for assessing reasoning skills, 
were taken from the Video-Taped Group Test (VTGT), which was developed for 
assessing cognitive levels of Israeli secondary school students (Shemesh, 
1983) • The test is based on 12 video-taped tasks, which were taken, with 
necessary adaptations, from i^wson*s test (1978). In each task, the 
video-taped demonstration served as prop for raising questions to the 
observers. The test has an internal reliability of .83 and inter- judge 
scoring agreement of 9 It. Its content and construct validity were reported 
Shemesh & Lazarowitz (1984), A schematic presentation of VTGT tasks is 
displayed in Appendix A. 

Scoring ; For each correct answer and explanation, students received 2 
points; for a wrong answer but a correct explanation - 1 point, zero points 
for the other two possibilities. Thus each student was eligible to receive 
a score from 0 to 21 points. 

Proceduret As the effect of each independent variable was studied 
separately, the specific design of each experiment and its results will be 
presented and discussed separately. 
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Experiment 1 - Method of Task Presentation; 

In this experiment the tasks were presented to subjects In two methods: 1) 
Vloeo-taped group demonstration (V); 2) written questions with graphic 
Illustrations (W). Examples of two tasks with W mode of presentation are 
displayed In Appendix B) • 

The effect of method of task presentation on the level of task performance 
was analyzed by age and gender. Each age group consisted of two Intact 
classes, which were randomly selected from the appropriate grade level, 
thus forming a 3 x 2 x 2 research design (7th grade, N =71; 9th grade, N = 
64; 12th grade, N = 40) x (V; W) x (boys and girls). 

Mean scores, standard deviations and t-test comparisons of the findings are 
presented In Table !• 

INSERT TABLE I ABOUT HERE 
As for a possible Interaction effect among age, gender and method of task 
presentations, students' scores were treated Dy a three-way analysis of 
variance. Results are displayed In Table II, 

INSERT TABLE II ABOUT HERE 
The results of the analysis of variance Indicate that age and gender were 
the main contributors to the variance In students' performance, but method 
and the Interaction between method and age had also significant 
contribution. There was no Interaction between method of presentation and 
students' gender. 

The results of this experiment showed that 7th graders performed 
significantly higher on the VTGT than on the wnitten version of this test. 
This effect was not found with 9th and 12th grade students. 
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Experiment 2 Format of Questionnaire : 

In this experiment the questions which follow each demonstration were of 
two formats: 1) Multiple choice questions with requirement for essay 
Justification (M); 2) open questions which required short essay answers 
(E). Examples of M & E formats are displayed In Appendix C. In each grade 
level, students from 2 classes were randomly assigned to one, out of two 
formats of questionnaires, thus forming a 3x2x2 research design (7th grade, 
N = 73; 9th grade, N = 72; 12th grade, N = 56) x (M;E) x (boys and girls). 
All students. In all grades, watched the same video-taped demonstrations 
and their questionnaires were followed by the same Illustrations, Mean 
scores, standard deviations and t-test comparisons of the findings are 
presented In Table III. 

INSERT TABLE III ABOUT HERE 
Three-way analysis of the variance (grade x gender x format of 
questionnaire) of students' scores with the two formats of the 
questionnaire are displayed In Table IV. 

INSERT TABLE IV ABOUT HERE 
The results showed that format of questionnaire had no effect on students* 
performance on the VTGT at least In grades 7tht 9th and 12th. 

Experiment 3 - Task's Numerical (k)n 

In this study, the effect of the numerical content of two quantitative 
tasks from the VTGT (proportions and probability) was studied. For this 
study a new version of the VTGT was created. In which the numerical values 
of the variables of two tasks were changed: 1) In .he task "Trees and 
Shadows" the non-Integer ratio (3:5) was changed to Integer ratio (1:2). 
2) In the task "Wooden Squares" the non-Integer ratio (2:3) was changed to 
Integer ratio (1:2).. The other 10 tasks were Identical In the two VTGT's 
which were administered to the subjects, therefore, only the performance In 
these tasks was analyzed. This analysis was performed by comparing the 
^ percentage of students' correct answers on the different versions of the 

EKJC 

two tasks, and are presented In Table V. 7 



INSERT TABLE V ABOUT HERE 
Findings in Table V showed that the type of ratio (integer/non-integer) 
between variables had significant effect on students' performance in all 
grades in the sample. While the non«-integer ratio items were treated with 
the concrete additive approach, even in the 12th grade, the integer ratio 
items were treated with proportional strategy, by 55>, 80% and 92% of 
students in grades 7th, 9th and 12th respectively. 

Discussion 

The results of this study indicate that there is an interaction between 

subjects' age and their reaction to various characteristics of formal 

reasoning tasks. While the format of the test questionnaire did not 

influence students' achievement in any age, the method of tasks' 

presentation affected performance of young subjects (grade 7th). Finally, 

the mathematical content of the quantitative tasks influenced students' 

performance in all ages. % 

Since the interactions betweem age and response differed by variables, each 

one will be discussed separately. 

1 . The impact of the method of tasks' presentation 

The results of the experiment in which students' achievement were evaluated 
in relation to the methods of task presentation showed that the method had 
effect only on 7th graders who performed significantly higher in the 
video-test than in the paper-and-pencil test only. These results can be 
explained by the assumption that the majority of young students (12-13 
years old) are in the level of concrete operational thinking and their 
abstraction ability is still very low. The video method of presentatir 
provided students with the necessary concrete examples and by that helped 
them overcome lack of abstraction ability. The older students who have a 
higher ability of abstraction did not need the demonstration. They could 
understand the nature of the tasks just by reading them. 

It should be mentioned that the use of a video method of tasks presentation 
motivated students in all grades to perform the tasks. 



2. The effect of questionnaire format 

Analysis of students' answers on the two different format of questionnaires 
have shown that the multiple-choice format has introduced no bias to the 
students. Warren (1979) found that it is easier to receive high s.iores on 
multiple-choice tests than in open questions* tests. This was not the case 
in this study, since the fact that students were nequired to justify their 
choice has diminished the guessing factor. 

3. The effect of task's numerical content 

Task's numerical content was found to have an important influence on the 
strategies which students, of all ages, used while solving proporton and 
probability problems. When integer ratio was used, most students succeeded 
to identify the correct solving strategies. When non-integer ratio was 
used, most students Jsed primitive (additive) strategies. Similar results 
were found by Hart (1980) who tried to identify hierarchial levels in the 
development of mathematical concepts. She found thai while 1:2:3 ratio 
problems were introduced, more than 50 percent of the students succeeded in 
solving the problems, compared to less than 30 percent of success with 
2:3s5 ratio problems. 

According to Inhelder (1977)» from 12 years and up, children can solve 
probability problems by using combinatorial and proportional operations. 
Accepting Inehlder's analysis as valid, students* performance on the 
probability tasks is depedent on their ability to identify the ratio 
between the target sample and all the sample. In other words, .students' 
' failure to perform the probability task results from their inability to 
identify the functional relationship which exists in the task and not from 
their misunderstanding of probabilistic concepts. The high percentage of 
correct responses to the 1:2 probability tasks provide support to the above 
thesis. 
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Impl 1 c a t Ion for sc 1 ence ed uc at 1 on 

Although the reasoning tasks used in this study were not in the borders of 
a specific science subject matter, the results have an implicaton in high 
school science education: 

a) The i mportance of ooncretizat ion in the instructional ^ process of ^'ounB 
students. This study re-emphasized the need of using concrete 
demonstrations while introducing concepts and principles in science 
instruction. Although the need for concretization decreases with students' 
maturation, still demonstrations may play an important role when students 
are exposed to new and unfamiliar subjects. 

b) The importance of the mathematical ^ aspects behind ^ a^ scientific problem* 
It is well accepted that mastery of many science concepts learned in high 
school required mathematical background (for example mechanics problems in 
physics; equations in chemistry, or Mendelian genetics in biology). 
Sometimes the mathematical aspects of a subject matter and not the concepts 
themselves cause misunderstan<?ing of scientific concepts. While familiar 
mathematical content such as the 1:2:3 ratio may facilitate the 
intemalizaton of formal strategies of problem solving, unfamiliar 
mathematical content (3:5:7) could draw students to adopt primitive 
strategies (as for example the additive approach instead of the 
proportional one in our experiment). Science concepts have enough inherent 
difficulties for the inmature learner and we better not confuse 'tudents by 
increasing those difficulties by using unfamiliar mathematical content. 
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Implications for fonmal reasoning test construction 

While Piagetlan clinical methods of assessment (Inhelder & Plaget, 1958) 
enable a deep study of chlldrens' modes of reasoning, group tests enable a 
vast survey of student mastery of different reasoning skills. The use of 
demonstration Is Importan" especially for young students (12-15 years old) 
who begin the transition from concrete to formal operational reasoning. On 
the other hand the use of the video-taped demonstrations restricts 
students' assessment to schools which have video-recorders. Since the 
performance of reasoning tasks by older students la not affected by the 
method of tasks presentation, demonstrations can be omitted when students 
above 15 years old are tested. Maybe It will be worthwhile to develop 
paper-and-pencll tests, which Include more tasks In each operation for 
these students. These tests should Include tasks with different levels of 
difficulty to Increase tests' discriminative power. The analysis of 
students' answers will enable the teacher or the Investigator to better 
understand the reasoning patterns of his students. From a methodological 
point of view, it Is desirable that researchers who study students' 
reasoning level will report In details the tasks and methods which were 
used. This Information will make possible the comparison between the 
results of different studies and by that help clarify the whole Issue. 
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Table I 



Means I Standard Deviations and ^-Values 
of VT6T Scores » by Method 



Grade 



Method 



Mean 



SD 



t-Values 



7 th 



9 th 



12th 



V 


38 


6.89 


?.06 


H 


33 


4.94 


2.66 


V 


35 


10.80 


4.12 


H 


29 


10.10 


5.11 


V 


20 


15.65 


5.07 


W 


20 


16.85 


4.71 



2.78* 



0.61 



0.77 



V t Video-taped demonstrations. 

W : Written questions with illustrations. 

* : p<0.01 
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Table II 



Three-way analysis of variance of students* scores 
on the VTGT» by grade » gender and method 
oli task presentation 



Source DF SS F P 



Grade 


2 


2721 


90.6 


0.0001 


Gender 


1 


288 


19.22 


0.001 


Method 


i 


29 


3.53 


9.052 


Method X Grade 


2 


75 


5.46 


0.001 


Method X Gender 


1 


25 


1.72 


n.8. 



r - 0.58; N - 175 
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Table III 





Means , S 


tandard 


Deviations and t - 


-Values 


of 




VI GT 


Scores, 


by Questionnaire 


Format 




Grade 


Format 


N 


Mean 


SD 


t-Values 




H 




6 flS 

o ♦ o J 






7t-h 

§ LIl 










n 1 

U . 1 J 




E 


38 


6.76 


2.42 






n 




in 9n 


^ • JO 




7 LIl 














E 


37 


9.45 . 


3.57 






M 


28 


13.39 


4.68 




12th 










0.26 




E 


29 


13.71 


4.51 





M : Multiple choice questions. 
E : Essay answers. 
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Table V 

Percentage, of correct answers to different types 
of proportional and probability Items 



Item 


7th(N-62) 
Z 


9th(N-71) 
X 


12th(N=48) 
X 


Proportions 
(3:5) 


5 


23 


72 


Proportions 
(1:2) 


55 


80 


92 


Probability 
(2:3) 


7 


30 


64 


Probability 
(1:2) 


72 


85 


100 
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Table IV 

Three-wa^ analysis of variance of students* 
scores on the VTGT, by grade, gender 
and format of questionnaire 



Source 


DF 


SS 


F 


P 


Grade 


2 


1703 


81.3 


0.0001 


Gender 


\ 


368 


35.2 


0.001 


Format 


1 


60 




n« s • 


Grade x Gender 


2 


9. 


0.45 


n*s. 


Format x Grade 


2 


13 


0.61 


n.s. 


Format x Gender 


1 


4 


o.;2 


n.s. 


r - 0.52; N 


- 200 
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Appendix A 
Schematic description of VTGT tasks 



Conservation 


Probability 

mmmama 




VioltiiM displacement 

e u 


Probability 




Propi 


ortlon 
1 




Combinations 


Proportions 


f 


Combinations 


Control of 

m 


variables 

\ 


Correlatlms 


Control of variables 


Correiations 
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APPENDIX B 



Two Examples of wriltten questions with Illustrations (H mode of task 
presentation)! 



Item 7i I put 4 nec squaties and 2 yellow squanes Into a sack. 
With closed eyes, I pulled out one square. 
What aiie the chances that this square Is ned? 



p ■ □ ■ □ ■ 

r tr \t tr V. ir 



Item 9t Suppose you have fouti different kinds of food (A,B,C and D). 
How many diffenent meals can you combine friom these foods? 



^Ifl^ 



6 
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APPENDIX C 



Examples of the Multiple-choice (M) and short essay answers (E) formats of 
quest Ions « which followed the VTGT demonstrations: 

H Format 

Item 7t Squares H H D H [1] I 

^ ^ "3 1i r 

What are the chances of pulling a 
red square out of the sack? 

□ 1 out of 2 p 2 out of «l 
P 1 out of i| p 2 out of 6 

□ 1 out of 6 a out of 6 

Explain youn answetis ' 



E Format 

Item 7 1 Squares 

-r r {J r i, I. 

Vniat are the chances of pulling a 

red square out of the sack? 

Explain your answertt 



M Format 

Item 12: Flowerj^ 

Is there any relation between the colour and 
the size of the flowef^s in this sample? 

15 3 1 9 

No, most of the big flowerls are white. 
Yes, most of the big flowens are white. 
No, there are flowena of all kinds. 
Yes, there arte flowens of all kinds. 
Explain yourt answers 



□ 
□ 

a 

D 



E Format 



The same stem, but without the fouti alternatives. 
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